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BRUSH FOR WARPING AND WATER REFLECTION EFFECTS 

BACKGROUND 

The present invention relates to computer- implemented methods for brush-based 
warping including causing reflection effects in an image. 
5 Computer painting programs allow the user to manipulate an electronic image with an 

onscreen "brush" controlled by an input device such as a mouse. One such program is Goo, 
available from MetaTools, Inc., 6303 Carpinteria Ave., Carpinteria, CA, 93013. 

The user may typically select from among a variety of brush shapes, e.g., circle, 
square, or single pixel. The user drags the brush along a "trajectory" from one location to 
1 o another within the image, thereby defining an "envelope." Pixels within the envelope are 
affected by the operation of the brush. 

One type of brush allows a user to "paint" colors and textures into the envelope. The 
paintbrush replaces the pixels in the envelope with the image of the brush. 

Another type of brush allows the user to "move" portions of the image within the 
1 5 envelope, creating a warping effect. For example, a "sticky" brush moves pixels from the 
starting point of the envelope to its end. A "slipping" brush moves the pixels from the 
starting point a distance less than, but proportional to, the length of the trajectory. Other 
warping brush effects are bloat, pucker, and twirl. 

20 SUMMARY OF THE INVENTION 

In general, in one aspect, the invention features a method and computer program 
product for generating a warping effect in an image having a plurality of regions, each 
associated with a distortion vector. The method includes defining a path segment vector in 
response to user movement of a brush cursor; modifying one or more distortion vectors 
25 having originations covered by the cursor based on the path segment vector; and repeating 
defining and modifying to produce a warping effect. 

Particular implementations can include one or more of the following features. 

The method includes sampling a source image using the distortion vectors to produce 
a destination image. 
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Each distortion vector ends at a mesh point in a distortion mesh, and modifying 
distortion vectors includes multiplying the path segment vector by a predetermined matrix, 
thereby producing a product vector; interpolating each distortion vector between mesh points 
that surround the end of the product vector, thereby producing an interpolated vector; and 
combining the product vector and the interpolated vector. 

The direction of the path segment vector is opposite to the direction of movement of 
the brush cursor. 

Multiplying can cause the magnitude of the product vector to be substantially twice 
the magnitude of the path segment vector. 

Multiplying can cause the direction of the product vector to differ from the direction 
of the path segment vector by substantially 90 degrees. 

The interpolation for the distortion vector can be linear, bilinear or of higher order. 

Defining includes defining a plurality of path segment vectors in response to user 
manipulation of a brush cursor, wherein the magnitude of each path segment vector depends 
on its location within the brush cursor; and modifying includes modifying a distortion vector 
using the nearest path segment vector. 

The method includes dividing the image into a frozen area and a unfrozen area by a 
boundary in response to user input; moving the cursor from the frozen area into the unfrozen 
area in response to user input; and modifying the distortion vectors in the unfrozen region 
covered by the cursor to produce a reflection effect based on a portion of the frozen area. 

The distortion vectors form a distortion vector field, and the method includes 
generating one or more further distortion vector fields; modifying each distortion vector field 
using a different time- varying function; combining the resulting distortion vector fields to 
produce a time- varying distortion vector field; and repeatedly sampling the source image 
using the time- varying distortion vector field to produce an image having an animated image. 

The sum of the time- varying functions is one at any given time. 

Each of the time-varying functions is continuous. 

Advantages that can be seen in implementations of the invention include one or more 
of the following. A mirror brush is provided that creates a reflection effect in an image. The 
reflection effect is warped to look like a reflection on a water surface with waves. The 
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warped reflection effect can also be animated with very little processing expense to produce 
an animated water reflection effect An artist can move the brush to produce different types 
of waves. To the artist, effect feels like "painting the waves" on which the image is reflected. 

The details of one or more embodiments of the invention are set forth in the 
accompanying drawings and the description below. Other features and advantages of the 
invention will become apparent from the description, the drawings, and the claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Like reference numbers and designations in the various drawings indicate like 
elements. 

FIG. 1 graphically depicts a process for generating a destination image. 

FIG. 2 depicts a process for producing a destination image by sampling or simply 
mapping from a source image plane to a color space. 

FIG. 3 presents a side view of the source plane and destination planes. 

FIG. 4A depicts a distortion in a "state of stability" for 2 mesh points, for a forward 
warping brush. 

FIG. 4B depicts a distortion in a "state of stability" for 3 mesh points, for a mirror 

brush. 

FIG. 4C depicts a distortion in a "state of stability" for 7 mesh points, for a minor 

brush. 

FIG. 5 presents an image resulting from the use of the warping mirror brush on an 

image. 

FIG. 6 is a flowchart depicting a process that generates a warping effect. 
FIG. 7 is a flowchart of a process for creating an animated reflection. 

DETAILED DESCRIPTION 

FIG. 1 graphically depicts a process for generating a destination image. A destination 
image 106 is associated with a distortion vector field 104. The vector field 104 includes a 
plurality of distortion vectors 1 12. Each distortion vector 112 begins at a destination mesh 
point 1 10 and ends at a source mesh point 1 14. 
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Each destination mesh point 1 10 is associated with a region in the destination image. 
The destination mesh points 1 10 collectively form a destination mesh 1 16. Each source mesh 
point 1 14 is associated with a region in the source image. The source mesh points 1 14 
collectively form a distortion mesh 1 18. The distortion vector field 104 is used to sample 

5 from regions in the source image to generate regions in destination image 106. Each region in 
the source and destination image contains one or more pixels. 

In one implementation, sampling includes simply copying the pixels from regions in 
the source image to regions in the destination image according to the distortion vectors. In 
another implementation, sampling includes applying a predetermined weight function to the 

10 pixels copied. In FIG. 1, each vector 110 points toward the upper left of its mesh point 
Therefore, the objects in the source image appear to have moved toward the lower right in 

the destination image. 

In one implementation the mesh is repeatedly updated as the brush moves over the 

image. This implementation is now described in detail. 

1 5 Referring to FIG. 2, a destination image is produced by sampling or simply mapping 

from the source image plane 204 to a color space 202. While this implementation is 
described in terms of image planes, the image can be defined on any sort of surface, such as a 
sphere, cylinder, and the like. When a user first applies a warping brush to the source image, 
he intends to create a deformation W of the source plane to a first destination plane 208, 

20 where W is a set of distortion vectors describing the forward mapping. The system 
implements the warping as an inverse mapping w from the first destination plane to the 
source plane. When the user applies the warping brush a second time, the user intends a 
mapping V from the first destination plane to a second destination plane 210. The system 
implements the warping as an inverse mapping v. Further applications of the warping brush 

25 are handled in a similar manner, using farther destination planes and inverse mappings. 
Mappings w and v can be implemented as distortion meshes, as described above. 

Inverse mapping w is represented as a vector w = (p,q) at each point (x, y) of the first 
destination plane, where p and q are the horizontal and vertical components of w, 
respectively. Then w is represented as: 



4 
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>v = (p(x,y),q(x.y)) 

Inverse mapping v is represented as a "path segment vector" v = (v*, v y ) at each point 
(x,y) of the second destination plane, where v x and v y are the horizontal and vertical 
5 components of v, respectively. The direction of the path segment vector is opposite to the 
direction of movement of the brush cursor. 

The system concatenates mappings v and w in the following manner. FIG. 3 presents 
a side view, sighted along the x-axis, of the source plane and destination planes. Vector v 
maps from a point (x,y) in the second destination plane to a point (x+v*, y+ v y ) in the first 
1 0 destination plane. Vector w maps from a point (x,y) in the first destination plane to a point 
(jc+/?, y+q) in the source plane. The coordinate {x,y) is the location of a mesh point FIG. 3 
depicts the general case where vector v does not end at a mesh point. 

Because vector v does not end at the mesh point where w is defined, the system 
interpolates to generate a vector w' that originates from the point (x+v*, y+v y ) between the 
1 5 mesh points in the first destination plane and describes the distortion at that point. The 
interpolation for the distortion vector can be linear, bilinear, or of higher order. 

In general, the vector w' is approximated by a Taylor series expansion in two 
variables (x,y). 



In one embodiment, the mesh points form squares, and so the mesh points are 
separated horizontally by a, and vertically by a. In other implementations, the mesh points 
can form rectangles, or can be arranged in more complex way. The Taylor series can be 
25 approximated up to the first order as 



W= w{x+v x ,x + v y ) 



20 




dx 2 2 + dy 2 2 



+ 



(1) 



d 3 wv x d 3 w vjvy d 3 w v x v y 2 | 9 3 >v v/ { 
dx 3 6 + cbc% 2 *dxdy 2 2 dy 3 6 
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we get bilinear interpolation. By adding ftuther terms in the Taylor series, we get 
higher order interpolations. 

1 5 The corresponding expressions involve more than two mesh points because of the 

higher derivatives, and are apparent to those skilled in the art. Also, other interpolation 
methods can be used and there is a wealth of literature on the subject. See for example G. 
Wolberg, "Digital Image Warping", chapter 5, for an approach focused on antialiasing. 

As the user moves the brush, the movement of the brush is regularly sampled. For 

20 example, the movement can be sampled every l/30 th of a second. As each sample is taken, a 
path segment vector is generated For example, a vector w is generated during the first 
sample, and a vector v is defined during a second sample. The system then concatenates 
vectors v and w, as described above to produce a new vector w, which will be concatenated 
with a vector generated during the next sample. The process repeats in this manner for each 

25 sample. 

Each brush stroke comprises a plurality of tiny brush movements, each referred to as 
a "sample." For each sample, a plurality of path segment vectors (referred to as a "path 



(2) 



Then the total distortion (p ' q *) is given by 

p . =v +p + p(^-,y)- P p(x,y + a)-p Q) 
a a 

,. = +q+ ^±±£^ Vx+ ^y±fhi Vy (4) 

' a a 

A better quality can be achieved at the expense of speed if more than linear in v terms 
are used (in the above Taylor expansion). For example by adding the next term in (1), 
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segment vector field") is generated. The path segment vector field is used to modify the 
distortion vector field. This process is repeated for each sample to produce a continuous 
effect. 

The process of generating a path segment vector field for a sample is now described 

5 according to one implementation. A path segment vector is generated at each mesh point 
under the brush cursor. The direction of a path segment vector is opposite to the direction of 
movement of the brush cursor during the sample associated with that path segment vector. 

The brush can have a brush pressure that is constant or varying. Brush pressure 
describes the way in which the strength of an effect induced by a brush varies within the area 

1 o affected by the brush. A brush having an effect of constant strength is said to have constant 
brush pressure. A brush having an effect of varying strength is said to have varying brush 
pressure. One example of a brush having varying brush pressure is a warping brush. One 
type of warping brush induces a stronger brush effect near its center and a weaker effect near 
its edges. Brush pressure can be implemented by basing the magnitude of the path segment 

1 5 vectors on their location within the brush cursor. 

As described above, a slipping brush "moves the pixels" from its starting point a 
distance less than, but proportional to, the length of its envelope. A slipping brush having a 
warping effect could move pixels near the center of the envelope further than pixels near the 
edge of the envelope. This is so because the vectors v are multiplied by the brush pressure, 

20 which is different at different points in the cursor. For example, the brush pressure can be 1 
at the center of the brush, and smoothly go down to zero at the edges of the brush. 

The system limits the effect of each brush sample stroke. In one implementation, the 
effect of each stroke is limited to the size of a mesh tile. A mesh tile is the area between four 
adjacent mesh points. Consequently, the increments added to w at each brush stroke are 

25 limited in size. The x- and y- components of vector w are kept smaller than a. After repeated 
brushing over one area, the distortion in that area reaches a state of stability, as described 
below. From the user's point of view we get a smear which looks like a small area that gets 
stretched and greatly magnified in the direction of brushing. In terms of inverse mappings, 
the reason for this is that all of the distortion vectors in a line along the direction of brushing 

30 are sampling from the same pixel. Next this effect is described in more detail. 
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In a matrix representation equations (3) and (4) become 



\~ /new 



V 9 >w 



a a 
q{x + a,y)-q l | gUjV + fl)-? 



(5) 



5 We define a new coordinate system x\y ' such that the x 1 axis is opposite in direction 

to the direction of the brush stroke and the y' axis is perpendicular to the brush stroke in the 
plane of the image. Now q' = 0 and vy = 0. Then equation (5) becomes (omitting the prime 
0 notation everywhere) 
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(6) 



Because the brushing is opposite to x, v x > 0, each brush stroke sample adds a little to 
/?. Assuming an initial state of w(x+a,y) = 0, equation (6) becomes 



15 



(7) 



Let dp = pnew - p and dv = v*. The system accumulates small distortions dp into a total 
distortion /? 



20 



dp 



(8) 



Here the sum of all samples of brush movements dv is v and the current p =p(v). 
Equation (8) has a solution 
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p = a 



\-e 



(9) 



10 



15 



From equation (9) we see that, as the user continues brushing at one single mesh 
point, p approaches the magnitude of the distance a to the next mesh point, but never exceeds 
a, and a state of stability is thus reached. In a similar manner, it can be shown that/? will 
approach a for any mesh point. Therefore, the difference in distortion between two adjacent 
mesh points in the state of stability is equal to the distance between the mesh points. We see 
that/? = a in the limit FIG. 4 A depicts this situation. A vector/? extending from a mesh 
point x to a mesh point x+a, thereby implementing the forward warping effect described 
above. 

For this reason, all of the distortion vectors in a line along the direction of brushing 
sample from the same pixel. This implements a forward waiping brush. The forward 
warping brush includes two variations: a slipping brush and a sticky brush. The sticky brush 
is described above. The slipping brush is similar to the sticky brush, but each distortion is 
multiplied by a constant less than 1 . 



20 



Forward Mirror Brush 

One equation for a forward warping brush is given above as equation (5). In one 
implementation, a modification of the above technique is used to create a minor reflection 
brush. The system described above is changed so that in Equation (5) "1" is replaced by "2" 
to yield: 
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Equation (8) then becomes 
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Here the sum of all sample vectors dv is v and the current p =p(v). Equation (1 1) has 
a solution 




From equation (12) we see that, as the user continues brushing at one single mesh 
point, p approaches the magnitude of twice the distance a to the next mesh point, but never 
exceeds 2a, and a state of stability is thus reached In a similar manner, it can be shown that 
the difference between the magnitudes of adjacent vectors having same direction will 
approach 2a. 

FIG. 4B depicts a vector p extending from a mesh point x to a mesh point x+2a, 
thereby implementing a reflection effect about mesh point x+a. We see that/? = 2a in the 
limit. FIG. 4C depicts a vector p beginning from a mesh point x-a and a vector pj beginning 
from an adjacent mesh point x-2a. We see that pi =p +2a, thereby implementing a reflection 
effect about mesh point x+a. 

Other effects can be attained by using other coefficients, A, B, C, D, which need not 
be integers. Equation (10) can be generalized as shown by: 
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The mirror effect discussed above is attained using 
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A perfect vertical mirror effect can be attained by eliminating the horizontal 
components, for example 



f A 
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Just stretching of the image by a certain amount can be achieved by equation (14) 
where "2" is replaced by a number less than 1. 

To achieve a more clearly defined reflection effect, in one implementation for each 
sample we execute the above calculation two or more times. This produces faster and better- 
defined saturation and stabilization into the mirror state. 

Other warping effects are achieved by selecting other values for A , B, C and Z). 

FIG. 5 presents an image 504 resulting from the use of the warping mirror brush on 
an image 502. A user has designated the upper portion of image 502 as a frozen area and the 
lower portion of the image as an unfrozen area, and has used the warping mirror brush to 
create a warped reflection of the frozen area into the unfrozen area. As is apparent in image 
504, the warping mirror brush can be used advantageously to create waves, producing a 
water reflection effect If desired, a user can use a smoothing brush to remove the waves. 
For example, the smoothing brush can provide a simple blur of the mesh. 

In one implementation a straight-line boundary is established, dividing the image into 
a frozen region and an unfrozen region. Distortion vectors in the frozen area are undisturbed 
by the reflection process, while mesh points vectors in the unfrozen area may be modified by 
the process. Brushing from the frozen area to the unfrozen area produces an interesting water 
reflection effect. Brushing perpendicular to the boundary produces a mirror- like effect, while 
brushing at an angle to the boundary produces a different artistic effect. The boundary to the 
frozen area is used as a starting point for the above process of modifying the distortion mesh. 

Left Mirror Brush 
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The process of modifying a distortion vector for a sample for a "left mirror brush" is 
now described. A path segment vector is generated at each mesh point under the brush 
cursor. The coefficients of the matrix in equation (14) are chosen to rotate the path segment 
vector by 90 degrees. Each resulting product vector has a direction perpendicular to the 

5 direction of movement of the brush, and to the right of the movement of the brush. The 
length of the vector is determined by the displacement of the brush since the last sample. 
This brush "displaces pixels" to the left of the brush to produce an interesting effect. 

Using this brush with the above mirroring engine produces a reflection perpendicular 
to the movement of the brush and to the left of the direction of brush movement This brush 

1 0 is useful because there is no need to define frozen and unfrozen regions, and the reflection 
effect can be produced by consecutive, parallel, overlapping brush strokes. 

FIG. 6 is a flowchart depicting a process 600. The process generates a warping effect 
in an image having a plurality of regions, where each region is associated with a distortion 
vector. The process defines a path segment vector in response to user manipulation of a brush 

1 5 cursor (step 602). The process then modifies one or more distortion vectors (the vector field) 
having originations covered by the cursor based on the path segment vector(step 604). As 
described above, each distortion vector has an origination at a destination mesh point 1 10. 
These steps are repeated as the brush moves to produce a warping effect. The above 
modification process is represented on FIG. 3 and related text. 

20 FIG. 7 is a flowchart of a process 700 for creating an animated reflection. A user 

manipulates a mirror brush to create two different distortion vector fields describing two non- 
identical reflection effects from the same source image (step 702). Each distortion vector 
field can be created using process 600, described above. 

The process applies a different time-varying function to each distortion vector field to 

25 create two distortion vector fields (step 704). For example, a sin 2 (t) function can be 

multiplied by the magnitudes of the distortion vectors for one distortion vector field, and a 
cos 2 (t) function can be applied to the magnitudes of the distortion vectors for the other 
distortion vector field. Of course, other time- varying functions can be used. However, they 
should sum up to 1, at least approximately. 
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The process combines the distortion vector fields to create a time-varying distortion 
vector field (step 706). For example, the vector magnitudes of the time-varying distortion 
vector fields can be summed for corresponding distortion vectors. 

The process samples the source image using the time-varying distortion vector field 
5 to create a destination image (step 708). The sampling process is described above. 

Process 700 can be varied in many ways. For example, more than two distortion 
vector fields can be combined to create the time-varying distortion vector field. The 
coefficients should still sum up to 1, at least approximately. As another example, time- 
varying distortion vector fields can be combined with non- time- varying distortion vector 

10 fields. Each of the above-described variations of process 700 creates an animated reflection 
effect. When a warping mirror brush is used to create one of the updated meshes, an 
animated water reflection effect is produced. 

The invention can be implemented in digital electronic circuitry, or in computer 
hardware, firmware, software, or in combinations of them. Apparatus of the invention can be 

1 5 implemented in a computer program product tangibly embodied in a machine-readable 

storage device for execution by a programmable processor; and method steps of the invention 
can be performed by a programmable processor executing a program of instructions to 
perform functions of the invention by operating on input data and generating output. The 
invention can be implemented advantageously in one or more computer programs that are 

20 executable on a programmable system including at least one programmable processor 

coupled to receive data and instructions from, and to transmit data and instructions to, a data 
storage system, at least one input device, and at least one output device. Each computer 
program can be implemented in a high-level procedural or object-oriented programming 
language, or in assembly or machine language if desired; and in any case, the language can 

25 be a compiled or interpreted language. Suitable processors include, by way of example, both 
general and special purpose microprocessors. Generally, a processor will receive instructions 
and data from a read-only memory and/or a random access memory. Generally, a computer 
will include one or more mass storage devices for storing data files; such devices include 
magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and 

30 optical disks. Storage devices suitable for tangibly embodying computer program 

13 



Attorney's Docket No.: 07844-420001 
Client Ref. No.: P384 



instructions and data include all forms of non- volatile memory, including by way of example 
semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; 
magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and 
CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs 
(application-specific integrated circuits). 

To provide for interaction with a user, the invention can be implemented on a 
computer system having a display device such as a monitor or LCD screen for displaying 
information to the user and a keyboard and a pointing device such as a mouse or a trackball 
by which the user can provide input to the computer system. The computer system can be 
programmed to provide a graphical user interface through which computer programs interact 
with users. 

The invention has been described in terms of particular embodiments. Other 
embodiments are within the scope of the following claims. For example, the steps of the 
invention can be performed in a different order and still achieve desirable results. 
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